In many large computer systems such as are found in banks, at credit card transaction processing centers, etc., huge amounts of data must be moved and stored. Typically, very large disk arrays are used to store the data and these disk arrays are connected to file servers. These type systems require of the server/disk connection high reliability, high speed, large throughput and large bandwidth since huge amounts of information are being processed and a system shutdown or slowdown adversely impacts customers (and possibly safety in the case of large air traffic control computer systems). Because these systems can be spread out over several buildings or even if all the servers and disk arrays are within the same building, the easiest way to interconnect all the servers and disk drives for maximum redundancy is through a local area network usually with a hub having drop lines connected to all servers, disk drives and other units. Because disks fail frequently and must be replaced by redundant disks, there is a large amount of management and control traffic that must go back and forth over the server/disk array connections and through the hub to the management and control process. This management and control traffic is necessary to determine which disks are on-line, where those disks are on the network, operational disk status, how many fans are still running (because when a fan fails, the disk it cools will probably be next), what servers are still operational, network fault status, network traffic conditions and statistics, etc.
Most if not all such digital data transmission systems require the bidirectional transmission of digital management and control data between nodes to collect data regarding the performance of the system and manage the various nodes, bridges etc. in the system. The out of band management and control data will be hereafter referred to as subchannel data.
There are many ways of modulating subchannel data onto a high speed digital signal, many of which are taught in a prior co-pending application of the assignee of the present application, which is hereby incorporated by reference. However, many of these prior art methodologies will only work in some systems and not in others. For example, modulating the subchannel data onto the clock by phase or frequency modulation will cause too much jitter in some systems, and also requires access to the high speed transmitter and receiver clock generation and clock recovery circuts. This is not possible in all cases, so a methodology that does not requires such access is preferred. The embodiments disclosed herein do not require such access. Amplitude modulation of the data with the subchannel modulation will not work in systems where digital buffers stand between the subchannel transmitter and the transmission media such as occurs where the high speed data media is fiber and a digital electronic-to-light media driver transducer is used to convert the digital high speed data to light signals.
Out of band management topologies get management data to and from the hub easily enough on separate network segments, which has its own set of problems described below. However, getting management and control data to and from the disk arrays in-band is more difficult. The management and control data sits in registers on a board in the disk array. In order to get this low speed, low priority data onto the high speed data path, special circuitry must be built in each disk array which interfaces these registers to the high speed data path. This circuitry functions to collect and format the management and control data into the type of data packets used in the high speed data path and to transmit these packets with the correct communication protocol to the hub. Since the management and control data does not consume an entire packet of the size used to send data on the high speed data path, some packet Space is empty and wasted. The need for this special circuitry to put the management and control data in the high speed data path makes the disk arrays more expensive and complex and placing management and control data in the high speed data packets wastes network throughput.
Some disk arrays store the management and control data on a separate disk drive which can be polled. However, this approach does not solve the problem of the need for special circuitry to get the management and control data into packets in the high speed data path and the resulting extra complexity and wasted throughput. It only allows time shifting thereby enabling transmission of the management and control data when network traffic volume is low and the throughput loss is not as significant. This time shifting is implemented by providing more storage capacity for the management and control data than is provided in the registers of the other prior art type disk arrays described above.
In some networks, management traffic is transmitted in-band by placing the management and control packets inside empty data packets and shipping these partially filled packets over the existing network connections between the server, hub and disk drives. This causes loss in throughput since the data packets are large and the management data does not fill the data packets entirely. This leads to wasted bandwidth. Further, arbitration by the management process to have one or more packets awarded to it for management traffic and to have access to the network so that transmissions to all servers and disk drive arrays of management messages consumes processing resources and network throughput unnecessarily.
One proposal has been made in the prior art to interleave special management and control packets in with the packet stream on the high speed data path to alleviate the above mentioned problem. However, this makes the design of the integrated circuits that implement the processing on the various network protocol layers more complex and difficult. So far this approach has been a commercial failure.
Another approach that has been tried in the prior art for transmission of management and control packets is to provide an entirely separate network for the management and control data such that each server and disk array is connected to the out of band management and control process running on a separate diagnostic processor by its own network segment. This substantially increases the wiring and connection cost of the system, especially in distributed systems, as every server and disk array must have an additional network segment connected thereto. In addition, each server and disk array uses a card slot for the management and control network card which adds to the expense, complexity and failure point count of each of these units.
A multiplexing approach that has been used in the prior art to send multiple television signals over the same media is represented by U.S. Pat. No. 3,623,105. This patent teaches receiving multiple video signals and translating each one to a different channel or subband and adding all the subband signals together to form a composite signal. The composite signal is then applied to the frequency control input of a VCO having a nominal frequency of 750 mHz. The output of the VCO is applied to one input of a pulse width modulator the other input of which receives an 18 gHz carrier. The pulse width modulator serves to key the 18 gHz carrier in accordance with the period of the signal from the VCO so that what is transmitted is a train of 18 gHz waves the width of each wave or “pulse” being set by the instantaneous period of the signal output from the VCO.
Another approach to implement a subchannel or auxiliary channel over a digital communication system is represented by U.S. Pat. No. 4,079,203 to Dragoo. This patent teaches an auxiliary channel implemented on a time division multiplexed carrier system by modulating the pulse repetition rate of the digital bitstream on the transmit side of the transaction. Each transceiver transmit section includes a FIFO shift register acting as a buffer. Modulation of the auxiliary channel data is carried out by varying the rate at which the digital information of the main channel is clocked out of the FIFO shift register at the transmitting end. The modulating signal varies the pulse repetition rate of a voltage controlled multivibrator which has its output coupled to the clock out input of the FIFO shift register and to a phase comparator of a phase lock loop. Data is clocked in using a clock signal derived from the incoming serial data stream. One drawback of this system is that the capacity of the FIFO can be exceeded if the clock out rate falls substantially behind the clock in rate because of the nature of the modulating signal during certain intervals.
Another approach which has been tried in the prior art is represented by U.S. Pat. No. 4,425,642 to Moses et al. This patent teaches sending digital data simultaneously with analog signals over the same media used by an analog signal communication system such as a telephone or television video. This is done by converting the digital data into very low power multifrequency signals consisting of fundamentals and harmonics. Although the harmonics are in the frequency range of the main signals, their low power does not cause substantial interference. Complicated filter arrangements at the receiver end separate out the harmonics and fundamentals that encode the digital signals and demodulation circuitry at the receiver decodes these frequencies back into digital data. This approach is not well suited to a system where the main data flow is digital and requires complex filtering and decoding arrangements at the receiver and is therefore unduly expensive and complex.
Another approach that has been tried in the prior art is exemplified by U.S. Pat. No. 4,677,608 to Forsberg. This patent teaches a method of implementing a service channel over a fiber optic system line which carries high data rate, e.g., 34 megabits/sec, data encoded with a code the power spectrum of which is heavily suppressed for low frequencies. The low frequency section of the media bandwidth is used to transmit a service channel with a low bandwidth. The service channel signals are frequency modulated, and this frequency modulated signal carrying the service channel data is then used to pulse width modulate pulses forming the high speed data stream.
It is critical to note in fully understanding the invention that this Forsberg scheme only transmits high speed data in the RZ or return to zero format on the fiber optic media, and, if NRZ format high speed data is input to the pulse width modulator, the NRZ format high speed data is converted to RZ format high speed data for transmission on the fiber optic media by performing an AND operation with the clock, as shown in FIG. 5 (see Col. 3, lines 58-61). Either way, the RZ data is a stream of pulses (during logic 1 bit times only) that are pulse width modulated by the subchannel carrier signal. RZ format high speed data is not compatible with standard high speed data receivers designed in accordance with the Fibre Channel and Gigabit Ethernet standards which are the network environments in which the various species of the invention disclosed herein are intended to work. Also, with RZ data, the number of transitions from one logic state to another is approximately double the number of transitions of NRZ data over the same number of bit times. Forsberg's system pulse width modulates each pulse representing a logic 1 so he gets perturbations of the time of crossing a reference voltage for both a rising edge and a falling edge of that pulse. With RZ data, if two consecutive logic 1's are followed by two consecutive logic 0's over four bit times, there will be a pulse with two transitions in each of the first two bit times (representing two consecutive logic 1's) and no pulses in the next two bit times. In contrast, with NRZ data for the same hypothetical, there will be a rising edge at the beginning of the first bit time and a falling edge at the end of the second bit time and no transitions during the third and fourth bit times.
The critical distinction is that NRZ data has a transition density which is variable over time depending upon the content of the data even if the data is D.C. balanced, while D.C. balanced RZ data does not have a variable transition density. RZ data is also incompatible with Fibre Channel networks. Since the subchannel information in Forsberg's system is expressed in the form of perturbations of the time of crossing the reference voltage on each transition of all logic 1 pulses, his system has a known and reliable number of transitions in which to send subchannel energy. Therefore, Forsberg does not have a variable transition density problem, and teaches no circuitry to deal with the fact that where pulse width modulation is in used to carry the subchannel information, transition density variations vary the magnitude of the subchannel signal at the receiver. Because RZ data is used exclusively in the Forsberg system as the data format on the media, a much stronger and nonvarying amplitude subchannel signal is available at the subchannel receiver which is easier to detect in the presence of noise.
Forsberg's system apparently was designed to work in telephony systems built by Ericsson (the assignee of the Forsberg technology) which are not transmitting One Gigabit digital data on a local area network. This conclusion can be drawn from the teachings at Col. 2. line 65 where the subchannel data rate is indicated to be 0.3-4 kHz, and Col. 3, line 6 where the FM modulated subchannel data signal is indicated to be 35-65 kHz, and from Col. 2, line 54 where the high speed bit rate is taught to be f0 and Col. 3, lines 21 and 22 where the clock rate is taught to be f0=1/T and Col. 3, lines 35-36 where the clock frequency is taught to be around 40 MHz.
The prior art subchannel transmitter described in co-pending application Ser. No. 09/063,633, filed Apr. 20, 1998, entitled SUBCHANNEL MODULATION SCHEME FOR CARRYING MANAGEMENT AND CONTROL DATA OUTSIDE THE REGULAR DATA CHANNEL solved these problems by using a form of pulse width modulation implemented by summing the high speed data with frequency shift keyed subchannel data modulated onto a carrier having a frequency that is much lower than the 1 GB high speed data. The frequency shift keyed carrier signal instantaneous amplitudes were added to the 1 GB data stream, but since the carrier frequency of the frequency shift keyed subchannel carrier was so much lower than the 1 GB data rate, the effect was as if a reference voltage level from which the logic 1 and logic 0 levels were measured was being changed at a slow rate relative to the high speed data rate. This raised and lowered the logic 1 and logic 0 levels of the high speed data in accordance with the lower frequency subchannel data, but these raised and lowered logic 1 and logic 0 levels were lost when the high speed data signal was passed through a digital buffer. However, the raised and lowered levels also changed the timing of zero crossings of the high speed data, so the subchannel data survived the digital buffer as pulse width modulation in the form of the perturbed timing of the zero crossings of the high speed data. Low pass filtering and pulse width demodulation techniques are used in the receiver to recover the subchannel data since the carrier frequency of the subchannel lies in the range of frequencies where the spectrum of the high speed data does not contain much energy since 8B/10B encoding or other DC balanced encoding schemes are used for the high speed data.
While this system works well, receiver sensitivity can be improved, and the subchannel transmitter and receiver can be made both smaller and cheaper by using digital techniques. The original subchannel receiver was relatively inexpensive since it was constructed with standard off the shelf ceramic filters that had a passband of about 110 kHz. Since these were readily available, they were inexpensive. However, that bandwidth is vastly greater than is needed to receive the relatively slow baud rate of the subchannel data. Thus, a need arose for a receiver design with a more narrow bandwidth that could be made more selective so as to reject more of the spurious noise in the frequency range of the subchannel signal. Narrower receiver passband characteristics allows subchannel signals with worse signal-to-noise ratios to be received.
Also, the original subchannel transmitter used an expensive numerically controlled oscillator and a high speed digital-to-analog converter and was not a one chip design so it was relatively expensive to build. The reason this particular analog design was selected was for flexibility purposes because it allowed the subchannel carrier frequency to be programmed. At that time, it was unclear what the subchannel carrier frequency should be to get best performance in the Fourier spectrum of the 8b/10b encoded high speed data. Certain GBICs and other third party vendor equipment that the subchannel signal had to pass through would attenuate the subchannel carrier more than other equipment. This was a problem because the subchannel carrier signals does not have much more power than the frequency components in the same frequency range caused by the 8b/10b encoded high speed data signal. The signal power of the subchannel carrier signal cannot be increased at will to improve the signal-to-noise ratio, because to do so causes greater pulse width modulation excursions which are interpreted in the high speed data receivers as jitter. Excessive jitter in the high speed data receivers can cause an intolerable bit error rate, and, in particularly bad cases, can cause loss of clock synchronization.
However, experience in actual operations in the field has brought to light the best frequency for the subchannel carrier. Further, the frequency deviation between the Mark and Space carrier frequencies that propagate best through the third party vendor components in the high speed signal path is better understood. Thus, there is a need for a subchannel transceiver design which can also be made both less expensive and smaller. Preferably, this subchannel transceiver is implemented as a one chip digital design with no programmability of the subchannel carrier frequency but a single chip is not absolutely required. Also, lack of programmability of the subchannel carrier frequency is not essential to the invention, and transceivers with programmable carrier frequencies are intended to be within the scope of the claims appended hereto.
Further, with the prior subchannel transceivers, much attention had to be paid to adjusting the transmit levels of the analog frequency shift keyed subchannel signal to avoid causing excessive jitter and to compensate for manufacturing variations in the other components used in the high speed signal path. This need for scrutiny and adjustment for every installation is inconvenient and is labor intensive. Thus, a need has arisen for a subchannel transceiver structure with less of a requirement for adjustment at the site of the manufacturer of equipment into which the subchannel transceivers are included. However, since there are many analog systems still deployed in the field, any new subchannel transmitter and receiver structures for the subchannel data must be compatible with these legacy systems.